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Cut-scores  are  commonly  used  in  industrial  personnel  selection, 
academic  selection,  minimum  competence  certification  testing,  and 
professional  licensing,  using  simple  and  mult iple-person/multiple- 
job  category  decision  paradigms.  Previous  approaches  have  proposed 
cut-score  solutions  in  a  variety  of  applications  (Cross  and  Su,  1975; 
Novlck  and  Lind  ley,  1978,  1979;  Petersen,  1976;  and  Van  dor  Linden 
and  Mellenbergh,  1977)  using  threshold  .  normal  ogive,  linear  and 
discrete  utility  functions.  This  paper  considers  these  results  by 
investigating  conditions  on  the  posterior,  likelihood  and  utility 
functions  required  for  a  cut-score  to  he  valid.  The  result  is  the 
showing  that  cut-scores  are  appropriate  in  a  wide  range  of  applica¬ 
tions,  but  they  are  less  than  universally  appropriate.  Following  this 
a  general  paradigm  and  computational  algorithm  for  cut-score 
solutions  is  developed  under  the  assumption  that  the  conditions  for 
a  cut-score  have  been  satisfied. 


I  nt  roduc 1  Ion 


In  .1  decision  problem,  .1  Bayesian  has  a  probability  distribution 
over  the  states  of  nature  and  a  utility  timet  ion  over  actions  and  states. 

A  Bayesian  makes  a  decision  by  maximizing  expected  utility.  Applications 
of  Bayesian  decision  theory  to  personnel  selection  have  recently  been 
studied  bv  Cross  and  Su  (197S),  Novlck  and  Petersen  (107b),  Novlck  and 
l.lndley  (19/8,  1979),  Petersen  (197b),  Petersen  and  Novlck  (1976)  and 
Van  der  Linden  and  Mellenbergh  (1977).  All  except  Novick  and  l.lndley 
(1978,  1979)  assume  utility  functions  are  threshold  or  linear,  and  the 
likelihood  functions  used  are  also  limited.  In  this  paper  wo 
study  personnel  decision  problems  using  more  general  utility  and  likeli¬ 
hood  functions. 

In  section  one  of  this  paper  we  discuss  the  likelihood  functions 
and  prior  distributions  we  shall  use.  We  show  that  the  stochastic 
increasing  (SI)  property  for  the  posterior  distribution  is  a  necessary  and 
sufficient  condition  fora  cut-score  for  an  arbitrary  utility  function. 

As  a  special  case,  we  study  likelihood  functions  satisfying 
monotone  likelihood  ratio  (MLR),  a  condition  studied  extensively  in 
Karlin  and  Rubin  (195b),  Karl  in  (1957a ,  1957b) , Lehmann  (1959)  and  DeCroot 
(1970).  Wo  define  the  monotone  posterior  ratio  (MPR)  property  and  show 
that  it  implies  the  stochastically  increasing  property)  thus  it  implies 
a  cut-score  for  an  arbitrary  utility  function.  We  also  show  that  MLR 
implies  MPR  with  any  prior  distribution. 

In  section  .two  we  note  that  most  posterior  and  predictive  densities 
do  not  satisfy  the  MPR  or  SI  properties,  but  show  th.it  in  important  cases 
cut-scores  are  appvoprinti  provided  that  discrimination  is  not  required 
in  the  tails  of  t h '  li str ibul ion.  In  particular  vr  carefully  study  the 
posterior  "t  1  distribution  obtained  with  many  Bayesian  normal  models. 
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In  section  three  we  provide  a  very  general  multi-person/ 
multi-job  category  paradigm  appropriate  for  a  wide  variety  selection, 
guidance  and  classification  applications.  By  assuming  the  appropriate¬ 
ness  of  cut-scores  based  on  the  theory  and  restrictions  of  section  2 
we  are  able  to  provide  simple  computational  algorithms  for  the  general 
paradigm. 

In  this  paper,  we  use  the  following  notation.  We  use  N  to  denote 
the  total  number  of  applicants.  0  is  the  parameter  we  are  interested  in 
and  T(Xj)  is  the  test  score  of  the  ith  applicant.  A  job  performance 
score  is  denoted  by  y..  In  one  context  we  use  A,B,C,....R  to  represent 
the  names  of  jobs  that  have  openings.  UA(0),  Ufi(0),  Uc (0 ) ,  . ...UR(0) 
are  the  utilities  of  assigning  an  applicant  with  ability  0  to  jobs 
A , B, C , . . .  and  R  respectively.  The  expected  utility  of  assigning 
a  person  with  test  score  T(x)  to  a  job,  say  A,  is  E (U^ (0 ) ) ,  which  is  a 
function  of  T(x).  The  indicated  expectation  is  with  respect  to  the 
Bayes  distribution  for  0.  We  let  VA(T(x))=  E(UA(0)).  This  notation 
emphasizes  that  V  does  not  depend  on  0 .  An  optimal  decision  procedure 
is  a  procedure  that  maximizes  the  sum  of  expected  utilities  of  the 
selection  across  all  persons  to  be  selected.  In  another  context  the 
utility  function  is  defined  over  the  performance  scores  y.  The  utilities 
are  then  U  (y),...,  U  (y)  and  expectations  are  taken  with  respect  to  the 
posterior  predictive  distribution  for  y.  All  results  are  valid  whether  the 
focus  is  on  the  unobservable  ability  (0)  or  the  observable  performance  y. 


Likelihood  Function  and  Prior  Distribution 

Let  0  be  a  person's  ability  (or  a  measure  of  performance),  and 
T(x)  be  his  test  score.  It  may  be  reasonable  to  desire  that  if  one 
person  lias  a  higher  test  score  than  another  then  the  statistical 
analysis' will  provide  some  evidence  that  the  first  person's  ability  is 
not  less  than  the  second,  nil  else  being  equal.  Thus  it  may  bo  reasonable 
to  desire  that  the  densities  p(x|0)  have  monotone  likelihood  ratio  (MLR). 

Definition  1:  The  real  parameter  family  of  densities  p(x|o)  is 
said  to  have  monotone  likelihood  ratio  if  there  exists  a 
real-valued  function  T(x)  such  that  for  any  *■'  t>2  the  likelihood 

functions  p(x|07)  and  p(x|0^)  are  distinct  and  the  ratio  p (x 1 0  ,) /p(x 1 0 j ) 
is  a  non-decreasing  function  of  T(x).  The  MLR  condition  is  a  particular 
technical  requirement  which  implies  that  increasing  values  of  T(x)  will 
provide  increasing  evidence  from  the  likelihood  that  0,  is  greater  than 
0j  when  in  fact  0 7  >  0^. 

It  can  be  shown  that  the  exponential  family,  (normal,  binomial, 
Poisson,  exponential,  etc.)  and  some  other  widely  used  distributions 
have  MLR.  This  condition  is  not  satisfied  by  all  distributions  (e.g. 
the  predictive  distribution  of  Student's  t  from  linear  regression); 
however,  in  such  cases  it  may  be  except  for  extreme  values  of  T(x). 

Monotone  likelihood  ratio  is  generally  an  easy  condition  toverify. 

From  a  decision  theory  point  of  view,  we  consider  0  as  a  random 
variable  and  define  a  similar  ratio  monotonicity  condition  for  such 


distr ibut ions . 


Definition  2:  The  class  of  posterior  densities  p(6|x)  is  said  to 


have  monotone  posterior  ratio  (MPR)  if  there  exists  a  real  valued 
function  T(x)  such  that  for  any  T(Xj)  <  Kx^)  the  posterior  distributions 
p(0  iKx^))  and  p(o|t(x^))  are  distinct  and  the  ratio  p  (0  |  T(xn ) ) /p(0  |t(Xj  ) ) 
is  a  non-decreasing  function  of  0. 

We  can  see  definition  2  is  exactly  the  same  as  definition  1  except 
the  roles  of  0  and  T(x)  are  exchanged.  The  suggestion  here  is  that  for 
any  prior  distribut ion,  higher  values  of  T(x)  will  be  suggestive  of 
higher  values  for  0  in  the  posterior  distribution.  The  following  lemma 
establishes  this  condition  for  MPR. 

Lemma  1:  Suppose  p(0)  is  any  prior  distribution  and  f(x|0)  has 
MLR  in  the  statistic  T(x),  then  the  posterior  distribution  p(fl|x) 
of  0  has  MPR  with  statistic  T(x). 


Proof:  By  definition  p(olx)  =  — ^  - 

I p(0)f (x | 0 ) dO 


Then  the  posterior  likelihood  ratio  is 


p(o|x2)  p(0)f(x2|0) 

P(0|x1)  ;p(0)f(x2|0)dO 

Jp(0)f(x1|0)dO 
ip(0)f (x2|0)d0 


ip(0)f (x. | e)de 
pfemxje) 
f(x2|e) 
f(x1|e) 


By  definition  of  Ml.R,  we  know  for  any  two 


values  0^  <  02  and  T(x^)  <  T(x^)  that 


h 


Since  Xj.x.j.flj,  and  0.,  are  all  fixed  we  can  write  this  as 

p(0llx2)  <  P<tf2l  X2} 

P(61|V  P(02|V 

and  conclude  that  p(0|x)  has  monotone  posterior  ratio. 

in  this  paper  we  generally  use  the  same  prior  distribution  for 
every  applicant.  This  is  appropriate  when  the  applicants  have  the 
same  kind  ot  background.  If  the  applicants  are  t rom  different  groups 
we  might  use  different  prior  distribution  for  the  applicants  from 
different  groups.  The  force  of  this  lenna  is  that  with  identical  prior 
distributions  an  ordering  on  T(x)  produces  the  same  ordering  as  would 
an  investigation  of  probability  ratios  from  the  posterior  distribution. 

While  MLR  and  MPR  are  useful  conditions,  a  weaker  condition  is 
adequate  for  our  purposes. 


. . .  -  — - 


- . -- 
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Definition  3:  A  posterior  distribution  is  said  to  be  stochast ical ly 
Increasing  (S!>  in  the  statistic  T  if  TlXj)  >T(x.,) 

and  for  all  6  we  have  FT(x^<®)<  KT(x  )(«).  where  F^  j(0)  and  FT(x  ^6) 
are  the  cumulative  posterior  distributions  of  tiwitli  observations 
KXjl  and  T(x^)  respectively. 

The  theorem  that  follows  is  the  foundation  of  this  paper.  We  will 
use  properties  of  this  theorem  throughout  the  paper. 


Theorem  1:  Let  g(P)  be  an  arbitrary  non-decreasing  function  of  0,  then 
E[g(0)|T(x)]  is  a  non-decreasing  function  of  T(x)  if  and  only  if 

the  posterior  distribution  is  stochastically  Increasing  in  T(x). 

Proof:  We  first  assume  that  T(x.)  >T(x.)  and  for  all  Owe  have  Fm/  .(0) 

1  “  2  T(x  )' ’ 

<  F 

—  Tfx^liO).  Proof  of  sufficiency  then  follows  the  argument  given  by  Lehmann 
(1959,  p.  I  la) .  We  now  extend  this  work  and  demonstrate  necessity. 

Assume  for  any  g(t’),  E[g(0)|T(x)J  is  a  non-decreasing  function  of 
T(x).  In  particular  we  assume: 


8(0) 


-1  if  0^8 

0  if  0>0*, 


then  F.(g(0)  |T(x)  ]  -  ~Ff(x)  (®  )• 

Thus  when  T(x.)^T(x2),  we  have  -F  ,  (8)>_-F  .(6*). 

*■  -  *  /  T (x^) 

In  above  0 


is  any  0 1  <JJX  thus  for  any  0  and  T(x.  )>_  T(x.,)  we  have  F_  (0)  < 

1  ^  — 


FT(x2)<9> 


A  posterior  distribution  satisfying  MPR  also  satisfies 
SI-  (Lehmanrt's  proof  for  MLR  requires  only  trivial  modification  for  this 
case.)  Thus  we  have  the  following  corollary: 

Corollary  1:  Suppose  f(::|0)  has  MLR  wilh  statistic  T(x)  and  g(0) 
is  a  non-decreasing  function  of  0,  then  E(g(6)|T(x)]  is 


a  non- 
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decreasing  function  of  T(x).  (Similarly  if  h(0)  is  a  non-increasing 
function  of  0,  then  E[  h(0)  |  T(x) ]  is  a  non- increasing  function  of  T(x)). 

In  application  g(0)  will  be  a  utility  function  U(6)  and  we  will  refer 
to  this  property  as  the  monotone  expected  utility  (MEU)  property.  The 
force  of  this  corollary  is  that  a  cut-score  (CS)  x*  can  be  set  is  guaranteed. 
For  if  0  has  MPR  and  T(x,,)iT(x  )  ^(x^, 
then  E(g(0)  |  T(x.,)J  <  E|g(o)  |  T(x  )  1  <  E[g(0)  |  Tfx^  ] . 

Thus,  in  summary,  we  have  for  an  arbitrary  utility  function 
MER-*  MPR  SI<i=>MEliO-  CS 

Cut  Scores  for  Specific  Models 

The  weakness  in  the  theory  of  the  previous  section  is  that  many 
posterior  distributions  are  not  MPR  or  SI.  Specifically  in  a  wide 
range  of  applications  the  posterior  marginal  distribution  will  be  "t". 

The  following  analysis  demonstrates  that  these  "t"  distributions  are 
not  SI  or  MPR  and  that  the  use  of  a  cut-score  is  not  necessarily 
justified  in  all  cases. 

Consider  the  simple  linear  regression  model, 
y  =  a  +  Sx  +  e 

The  value  a  and  0  are  respectively  the  intercept  and  slope  of  the  linear 
regression  function,  y  is  the  performance  ability  and  x  is  the  test 
score  which  is  the  same  as  t  in  the  previous  sections.  Following  the 


.,r, 
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development  of  Petersen  (1976)  we  assume  that  the  variance  of  the  error 
Is  homoscedast ic ,  i.e., 

i  2 

Var(y  [  x)  ■  o' 

for  all  x.  We  also  assume  that  e  is  normally  distributed  for  fixed  a, 6 
and  x. 

Suppose  that  u,ii  and  log o are  uniformly  and  independently  distributed, 
Civen  n  pairs  of  observations  (x,y)  the  posterior  predictive  distribution 

y  for  an  applicant  with  the  score  X  »  x^  has  a  Student  t  distribution 
of  (n-2)  degrees  freedom,  specifically,  (Novick  and  Jackson,  1979) 


given  X  -  xQ  , 


y  -  ly  +  *  (X0  ~  X)|  :  t (n-2) 

s[  n  +  1  (*0  *r’  ’1/2 


(1) 


n  "  +  E(x  -  102 


where  x  -  Ex/n  y  =  Ey/n  S'  =  i(x  -  x'),  S'  «  £(y  -  y)2. 


xy 


E(x  -  x)  (y  -  y) ,  d  =  S  /S'  and  s' 

xy  x 


(s2  -  s2  /S')/ (n  -  2). 
y  xy  x 


Let  y  be  the  minimum  level  of  satisfactory  performance  for  a  threshold 
utility,  then  for  a  given  test  score  X  =  x^  the  expected  utility  is 

EfU  (y)  |Xq]  *  Pr  (Y  >  y*  |  xQ) 

A 


1-Pr 


J  1  <  - 

\  n  +  1 


y  -  [y  +  s  (xQ  -  x)  l 


s[ 


,  (X0  ~  X):  ,1/2 

Z (x  -  X)2 


(2) 


where  t  is  the  standard  student  t  variable  with  (n  -  2)  degrees  of 
freedom . 


S 


'Air*  v 


- - . ■  -  •  - 
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To  see  that  the  posterior  predictive  distribution  of  the  student  t 
variable  in  (1)  does  not  have  SI.  Let 


«(xn>  -  -X  r.  (y  +.JLX»0  -  x)) _ 

.. 

n  l.(x  -  xi 

Equation  (2)  shows  that  the  expected  utility  increases  as  g(xg) 
decreases.  Therefore,  the  cut-score  can  be  set  If  E|U(y)  |  x^)  is  a 
decreasing  function  of  x„.  Taking  the  derivative  of  g(xn)  we  get 


*  -  (x0  '  x) 

-(v  -  y)  - U 


Z(x  -  x)2 


-3  n  +  1 
n 


hT~  8<V 


Setting  3 


n  4-  1  +  (X0  -  X)' 


3/2 


E(x  -  x) 2 


3  x. 


g(x^)  <_  0,  we  have 


,  .  -  (xo  -  x) 

■<y  -y) -orr^-,2 


"  n  +  1 

1 


(3) 


/  *  —  v  ,  — .  <•  n  +  1  £ (x  -  x)  (v  -  y) 

or  -(y  -  y)  (x  -  x)  1 -  V7  y> 

0  n 

as  the  condition  required  for  a  cut  score.  Consider 

★  — 

CD  y  =  y  ,  then  (3)  is  equivalent  to 
0  <_  E  (x  -  x)  (  y  -  y) 

In  this  case  a  cut  score  can  be  set  if  the  corrected  sum  of  cross  products 
(or  the  slope  of  the  regression  line)  is  non-negative.  Now  consider 

■k  — v 

ii)  y  i  y  then  (3)  becomes 


* 


0  - 


^  -  n  +  1  ,  E(x-x)  (y-y)  for  y  >y, 

^  X  “  Tfc  '  '  .  _ 


Y  ~  y 


(4) 


and 


xo± 


x  + 


n  +  1 
n 


? (x  -  x)  (y-y) 

y-y* 


for  y 


(5) 


. — . 
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Combining  (i)  and  (il)  for  any  arbitrary  threshold  utility  y  .  a 
cut  score  can  be  set  if 


-  n  +  1  E  ( x  -  x ) ( y  -  y)  1  x„  <  x  +  n  +  1  l(x  -  x)  (y  -  y) 
*  -  - - “*  -  0  n  I  yw  -  y| 


|y*~  VI 


.  -  |  r  n  +  l  Kx  -  x  )  (y  -  y) 

1*0  -  x  1  -  ~  y^-tr 


The  equation  (7)  can  be  expressed  as 


(6) 


(7) 


|xQ  -  xl 


-Li! —  y  <  (n  +  1)  r 
s 

y 


(8) 


where  r  is  the  sample  correlation  coefficient  for  (x,y)  and 


2  \  1/2 


and  sy  =  —  y- 

’  ’  n 


2  i  1/2 


This  shows  that  a  cut-score  can  be  set  unless  n  is  small,  r  is  small, 

★  —  . 
y  is  distant  from  y  and  there  are  x  values  very  distant  from  x. 

This  condition  may  be  satisfied  for  some  scores  in  the  case  of 

high  selectivity. 

The  1968  data  on  college  7  for  junior  college  test  scores  from 
the  American  College  Testing  (ACT)  Program  will  be  used  for  the  example. 
The  y  variable  is  the  first  semester  grade-point  average  (GPA)  and  the 
predictor  variable  x  is  English  test  score.  Since  most  colleges  require 


1 2 


a  minimum  2.00  for  graduation  wo  set  y  •  2.00.  The  original  data  contain 
105  observations.  For  the  purpose  of  the  example  we  picked  the  i  list 

10  observations.  For  n  -  105.  x  -  19.13,  v  -  2.28,  S‘  -  2766.13. 

x  * 

2 

S  *  84.34,  S  “  235.47,  6  •  0.085,  s  -  0.79  and  D  ■  849;  for  n  •  10, 
y  xv 

_  j  2 

x  -  22.1,  y  -  2.66,  -  140.9.  S  -  6.4,  Sxv  -  5.54,  6  -  0.019, 

a  "  0.88  and  D  -  9, 


whore 


0 


n+ 1  ' (x-x) (v-y) 
n 


Since  the  range  for  ACT  English  Scores  is  from  0  fit  36,  for  n  *  105, 

the  condition  in  (5)  is  satisfied  for  all  x0;  therefore,  a  cut  score  can 

be  sot  and  the  expected  ut  i  l  its  is  an  increasing  1  unction  of  the  test 

score.  For  n  ”  10,  the  condition  in  (b)  is  violated  for  x^  '  32;  hence 

setting  a  cut  score  is  Infeasible.  Table  1  contains  the  values  of  the 

expected  utility  fot  each  sample  for  x^  •  0(2)36.  For  n  »  103,  the 

expected  utilitv  is  increasing  in  test  score  x(>  ;  however,  for  n-10, 

the  expected  utility  increases  and  then  decreases  at  x(1  »  32,  Hence  a 
.lit -score  cannot  be  set. 


insert  Table  1  about  here 

It  is  possible  to  treat  \  more  general  class  of  utility  functions 

in  precisely  the  same  wav.  Suppose  that  the  utility  of  selecting  a 

2 

person  of  ability  y  is  a  normal  ogive.  Let  u0  and  oQ  be  the  mean  and 
variance  f-r  this  ogive  so  that 


.  .»j.  x _ at>., 

. . - 


b(y)  -  Cj) 


If  n  is  large  the  t  distribution  in  (1)  is  approximated  by  a  normal 


variable  with  mean  y  +  fc  (xq”  an<^  variance 

a2{ | (n  +  l)/n]+[(x0  -  x)2/  £(x  -  x)‘)}. 

The  expected  utility  (Novick  and  Lindley,  1979)  of  selecting  a  person 


Following  the  techniques  used  previously,  the  condition  for  which  the 


cut-score  can  be  set  is 


x0  "  x|  ]u  -  y  j  1  r(n  +  1)  +  r|_0 


This  result  may  be  compared  with  (8  )  noting  that  in  the  special  case 


gq  -  0  the  two  expressions  agree.  For  the  previous  example  (n-10)  (9)  Will  be 
satisfied  provided  oQ>0.58.  Threshold  utility  is  a  special  case  of 
normal  ogive  utility  with  Oq"0  and  thus  it  is  seen  to  be  the  "worst  case". 
This  may  generally  be  true  though  we  offer  no  proof  for  other  ogival  forms. 
In  a  recent  topical  issue  of  the  Journal  of  Educational  Measurement 


there  was  some  controversy  over  the  issue  of  whether  or  not  the  setting 


of  cut-scores  was  feasible.  One  paper  (Class,  1978)  was  very 
pessimistic,  another  was  very  optimistic  (Hambleton,  1978).  Others 


were  less  firmly  committed  on  this  point.  All  of  the  discussions  in 
these  papers  seemed  to  be  in  the  context  of  threshold  utility,  though 
it  was  not  always  apparent  that  a  clear  distinction  was  being  made 

A 

between  the  minimum  criterion  level  0  in  threshold  utility  and  the 

cut-score  x  in  the  observation  space.  Criticism  (Glass,  1978)  of  attempts  to 

* 

specify  a  threshold  cut  0  seem  valid  as  they  are  effectively  criticisms 
of  threshold  utility  which  Novick  and  Lindley  (1978)  clearly  suggest  is 
no  better  than  a  first  rough  approximation  to  a  realistic  utility 
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The  point  of  the  present  paper  is  that  cut-scores  generally  do 
exist  and  can  be  rationally  specified  to  the  extent  that  utility  functions 
can  be  accurately  specified.  They  do  not  depend  on  the  assumption  of 
threshold  utility.  Only  the  lesser  assumptions  ot  monotone  likelihood 
ratio,  stochastic  increase,  or  the  demonstration  of  monotone  expected 
utility  locally  for  the  particular  likelihood,  prior  distributions  and 
utilities  are  needed.  Thus  a  focus  of  work  ought  to  be  on  the  development 
of  coherent  bias-free  methods  of  assessing  utilities.  In  this  connection 
a  recent  discussion  by  Novick,  Chuang,  and  DeKeyrel  (in  press)  seems  relevant 
The  counter  argument  to  this  is  that  the  utility  of  selection  often 
cannot  be  related  to  available  criteria  and  that  the  more  important 
factors  needed  to  be  evaluated  less  formally.  This  paper  does  not 
contribute  to  the  debate  on  this  point  but  does  argue  that  other  problems 
are  manageable. 


f*  1 


♦ 
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Selection  for  a  Single  Job  Category 

In  this  section  we  consider  that  there  are  N  candidates  applying 

for  one  job  category  A,  that  has  openings.  We  will  study  optimal 

selection  procedures  for  quota  free  and  restricted  models. 

Suppose  job  A  has  openings  and  let  the  utility  of  accepting  someone 

with  ability  0  be  U^(0)  and  the  utility  of  rejecting  someone  with  ability 

0  be  UR(0).  In  the  case  of  personnel  selections,  the  greater  the  ability 

of  the  candidate  the  greater  will  be  his  value  to  potential  employers. 

We  may  assume  that  U  ( 0 )  is  a  non-decreasing  function  of  0  and  II  (0)  is 

A  K 

a  non-increasing  ,  constant,  or  more  slowly  increasing  function  of  6  than 
UA  ( 0 )  .  Then  the  marginal  utility  of  selection,  1)  (0)  “  bR(°)»  n 
non-decreasing  function  of  0. 

Theorem  2:  Under  the  conditions  of  theorem  1.  If  U  (0)  -  U_(0)  is 

A  K 

* 

a  non-decreasing  function  of  0  then  there  exists  a  cut-score  T(x  )  such 
that 

E[Ua(0)  |  T (x) ]  >E[Ur(0)  |  T(x) 1  for  T(x)  > T(x*) 

and 

E [ UA ( 0 )  |  T(x)  ]  iE[UR(0)  |  T(x)  ]  for  T(x)  <  T(x*) 

* 

In  extreme  cases  T(x  )  may  equal  or  +“>  . 

The  above  result  is  a  special  case  of  a  theorem  in  Karlin  and 
Rubin  (1956).  The  proofs  can  be  found  in  Lehmann  (1959,  p.  74).  The 
importance  of  this  theorem  is  that  the  existence  of  cut-score  is  derived 
rather  than  assumed.  Thus  in  the  case  of  monotone  posterior  ratio 
and  monotone  utility  functions  a  cut  score  can  always  be  used  to  maximize 
utility. 


/b 

Theorem  2  shows  that  if  the  applicant's  test  score  T(x)  is  greater 
then  T(x*)  then  the  expected  utility  for  acceptance  is  higher  than 
rejection  (hence  he  should  be  accepted).  This  is  equivalent  to  saying 
that  the  expectation  of  U^i.O)  -  UR(0)  is  non-negative,  that  is, 

EIV0)"  V0)  !  T<x>  1  for  T(x)  >  T(x*)  . 

Similarly,  we  have 

ElllA(rt)  -  Hr(0)|t  (x)  1  ^0  for  T(x)  <T(x*) . 

* 

Therefore,  the  cut  score  T(x  )  is  the  point  at  which 
E[Ua(0)  -  UR(0)  |  T(x)  )  changes  sign;  that  is,  E l UA ( °)  “  UR(e)  |  T(*)l 

changes  from  nonpositive  values  to  nonnegative  values  at  T(x  ). 

* 

In  particular,  if  E[U  (0)  -  U  (  0)  |  T(x)]  is  continuous  at  T(x  )  then 

A  K 

E[uA<e>  -  ye>  I  t(x*)]-  o. 

The  following  theorem  shows  that  under  our  conditions,  if  we  have  two 
applicants  and  exactly  one  opening  we  should  accept  the  one  with  the 
higher  test  score. 

Theorem  3:  Under  the  conditions  of  the  theorem  1.  If  UA(0)  ~  UR 
is  a  non-decreasing  function  of  6  and  T(xi>  >  T(Xj),  then 

VA(T(xt))  +  VR((x.))  iVA(T(Xj))  +  VR(T(Xi)), 

where  VA<T(x))  =  E[U  (8) |T(x) ]  and  VR(T(x))  -E[UR(0) |T(x) ] 

Proof:  By  Theorem  1,  E[U,(0)  -  Um(0)|t(x)]  is  a  non-decreasing 

A  K 

function  of  T(x).  Thus, 

Efiye)  -  UR(0)  |T(x,)]>  e(ua(8)  -  uR(e>  |t(x  )] 
so 

V^Tfx^)  -  Vr(T(x.))  >Va(T(x  ))  -  Vr(T(x.)), 

then 

VA(T(xi)  +  vR(T(xj))  iVA(T(x  ))  +  Vr(T(x.)). 

Assume  that  the  Lost  scores  . ■  1  N  applicants  are  T(xj),  Tfx,,),  ... 
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T(x^),  without  loss  of  generality  we  can  assume  T(x^)  >_T(x.,),  ... 

y  T  /  « 

-  N) .  For  job  category  A  we  may  have  three  situations: 

(1)  quota  free,  i.e.,  we  may  accept  as  many  applicants  as  we  want; 

(2)  we  want  to  accept  exactly  N  applicants  where  N.  <N; 

A  A 

(3)  we  want  to  accept  at  most  N.  applicants  where  N  <  N. 

A  A  — 

Note  that  if  ■=  N  then  (3)  specializes  to  (1).  The  decision  rules 
for  these  cases  are  very  simple.  From  Theorem  2  there  exists  a  cut- 
score,  say  C  ,  such  that 

V  (T(x))>V  (T(x))  for  T (x)  >C 


Va(T(x))<  Vr(T(x))  for  T(x)<C 


(a)  The  optimal  decision  procedure  for  situation  (1)  is  to  accept 
the  i-th  applicant  if  T(xi>  >  C  and  reject  him  if  T(xi)<  C. 

(b)  The  optimal  decision  procedure  for  situation  (2)  is  to  accept 
those  N  applicants  who  have  the  highest  test  scores  (i.e., 

A 

accept  those  who  have  scores  TCx^),  T(x2>  ...  T(xN  )). 

A 

(c)  The  optimal  decision  procedure  for  situation  (3)  is:  if 

T(xNa)>  C  then  accept  the  first  applicants,  i.e.,  the 

same  as  (b) ;  if  T(x^  )  <C  then  we  will  accept  applicants  whose 
A 


scores  are  greater  than  C,  i.e.,  the  same  as  in  (a) 


Proof : 


(a)  is  trivial,  because  we  maximize  our  utility  of  assigning 
each  applicant  ;thus  we  have  maximum  total  utility. 

(b)  is  a  direct  extension  of  Theorem  3.  If  we  exchange  our 
decision  about  any  two  applicants,  we  can  see  by  Theorem  3 
our  total  expected  utility  will  not  increase.  Thus  we  have 
optimal  decision  procedure. 


v  -**v 

■>  * 


(c)  is  a  combination  of  (a)  and  (b)  and  we  can  see  the  procedure 
is  optimal  immediately. 

Again  we  emphasize  that  this  theorem  is  generally  false  in  the 

absence  of  SI  but  the  use  of  a  cut-score  may  be  valid  with  reasonable 

practice.  We  also  note  that  while  we  assume  U  (0)  -  U  (0)  is 

A  K 

nondecreasing  in  0  we  arc  not  assuming  that  the  derivative  l/(0)  is 

A 

nondecreasing  in  0  .  In  particular  the  following  schematic  is  valid 
and  should  be  considered  typical. 
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Selection  for  Multiple  Job  Categories 

In  this  part  we  consider  that  jobs  A,  B,  C,  ...  R  have  openings. 

We  also  assume  there  are  N  applicants  and  the  utility  of  assigning 

someone  with  ability  0  to  jobs  A,  B,  C,  ...  H  are  U  (8),  U  (0)  ,  ...  1)  (0) 

A  B  K 

respectively.  In  general  these  kinds  of  problems  are  very  difficult.  We 
will  consider  an  important  special  case  of  the  situation  by  assuming 
C^(0)  ^  Ug(0)  •••  i  %<<»  *or  0*  (We  can  weaken  this  condition 
by  assuming  UA(0)  -  UB(0),  UB(0)  -  Uc(0)t  ...  1)^(0)  -UR(0)  are  all 

non-decreasing  funct ions  of  0.)  An  example  of  this  special  case  may  be 
that  job  A  is  group  leader,  job  B  is  system  analyst,  job  C  is  programmer 
and  job  R  is  rejection.  After  several  years  experience  a  programmer  may 
be  promoted  to  a  system  analyst  and  a  system  analyst  may  be  promoted  to  a 
group  leader.  Thus  jobs  A,B,C,  R  may  satisfy  our  conditions. 

The  following  theorem  shows  the  existence  of  the  multiple  cut-scores, 
a  result  in  Karlin  and  Rubin  (1956). 

Theorem  A.  Under  the  conditions  of  theorem  1.  If  UA(0)  -  Ug(0), 

11^(0)  -  Uj,(0),...  UR _^(0)  “  UR(0)  are  all  non-decreasing  functions  of  0, 
then  there  exists 


CB 

> 

. . .  >  <: 

K such  that 

if 

T(x) 

>  C 

A 

then 

Va(T(x)) 

"  V* 

(T  (x)  ) 

if 

C 

A 

>  T(g) 

>  C 

B 

t  hen 

Vb(T(x)) 

“  v* 

(T(x) ) 

if 

CB 

>  T(x) 

>  C 

C 

then 

Vc(T(x)) 

-  v* 

(T (x) ) 

(10) 


if  CR-1  T(x)  then  VR  V*(T(x)) 

where  (T(x))  »  Mnx^V-(T(x))  Vg(T(x)),  ....  V^(T(x)) 


{ 
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From  the  above  result  we  can  see  very  clearly  that  if  we  have  an 

applicant,  then  we  use  C^,  CR_j  as  cut-scores  and  decide  to  which 

job  to  assign  him.  For  example  If  T(x)  >  C. !  then  we  assign  thy  applicant 

to  Job  A,  If  C  >  T(x)  >  C  then  we  assign  the  applicant  to  Job  B  ...  etc. 

Intuitively  we  assign  a  person  with  higher  score  to  a  Job  that  gives 

higher  marginal  utility  (for  example,  job  A).  Generally  we  find  that 

C  Cp,...  CR  ^  are  distinct.  Thus  every  job  has  a  chance  to  receive 

the  applicant.  The  following  theorem  shows  how  to  find  CA>  Cg,...,  CR_j. 

Its  proof  can  be  obtained  by  repeatedly  applying  theorem  1. 

Theorem  5:  Under  the  conditions  of  theorem  4.  Assume 

E[Ua(6)  -  UB<0)  |  T(x)],  E[Ub(0)  -  Uc(fl)  |  T(x) ] .  E[Ur_i(0)  -  UR(0)  |  T(x) ] 

are  continuous  functions  of  T(x),  let 

K [ U  ( 0 )  -  U_(0)  |  T(x)  -  C.]  -  0 
A  D  A 

F. [ U B ( 0 )  -  Uc(0)  |  T(x)  -  CR]  -  0 

•  (ID 

I 

I 

E[Ur_i(6)  -  UR(0)  |  T(x)  -  C^]  -  0 

If  cx>  CB>  ,...CR  then  the  equation  (10)  holds.  Hence,  CA>  Cb**'-*cr_i 


are  the  cut-scores. 

For  the  conditions  C.  >C_  >  ...  >CD  .  in  theorem  5  ,  if  one  of  the 
equalities  is  violated,  then  the  corresponding  scores  for  the  solutions 
in  (11)  are  not  the  cut-scores.  Suppose  that  there  are  only  three  jobs, 

A,  B ,  and  C.  Let  C.  and  C_  bo  the  solutions  of  (11)  such  that  C  >C  ,  and 

lct  E(Ua(0)  -  Uc(0)  I  T(x)  -  cAC]  -  0 


(12) 

Then  C„  > C.  „  > C.  and 
3  —  AC  —  A 


if  T(x)  >CAC  then  VA(T(x))  -  V.  (T(x)) 

if  T(x)  iCAC  then  Vc(T(x))  -  V*  (T(x)). 
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Thus,  C  .  of  die  equation  (12)  is  the  only  cut-score,  and  job  B 
does  not  have  a  chance  to  receive  any  applicant. 

The  condition  U'(8)'  ll'  (e)  >. . .  >  1)'  (0)  or  the  alternative 

A  —  d  —  —  R 

U  (0)  -  U  (0),  U  (0)  -  tl  (0),  ...  IL  .(0)“  U„f0)  are  all  non-decreasing 
''BBC  R-l  k  0 

functions  ot  0  can  he  relaxed  to  include  important  additional  cases. 
Suppose  that  for  0>0  ,  U  ( t) )  -  U_(0)  is  a  positive  function 

1  A  u 

of  8;  for  8^>0  >  0^^,  Ug(8)  -  Uc(8)  Is  a  positive  function  of  8, etc. 

Suppose  further  that  a  multiple  cut-score  solution  yields  cut-scores 
C4.  and  C„  ,  then  it  is  evident  that  the  computed  solution  is  a  valid 

n  B  ■ 

solution.  Thus  we  see  that  the  required  monotonicity  properties  are 
local  rather  than  global  properties. 

The  monotonicity  assumption  on  utility  differences  in  the  weaker 
form  considered  above  seems  eminently  reasonable  in  both  employment  and 
educational  applications.  Consider  an  underlying  "general  ability, 8" 
and  the  occupational  careers  of  physicist  (A),  electrical  engineer  (B), 
and  electrician  (C) .  Then  the  utility  structure  pictured  in  figure  2 
may  be  valid. 

Insert  Figure  2  About  Here 
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The  following  example  demonstrates  the  use  of  cut-scores  in 
multiple  job  assignments.  Suppose  there  are  three  jobs.  A,  B  and  C 
are  available  and  suppose  that  the  utilities  of  jobs  A,  B,  and  C  are 

uA  (y)  . 

UB  <y)  y  - 2 . 2\ 

and 

»c<»> 

Using  the  1968  data  on  college  7  of  ACT  scores  given  in  section  2  , 
the  expected  utilities  for  each  job  are  given  in  column  2  to  column  4  of 
Table  2.  Column  5  is  the  difference  of  column  2  and  column  3,  and 
column  6  is  the  difference  of  column  3  and  column  4.  Column  5  and 
column  6  change  sign  at  x  =  30  and  x  -  24,  respectively.  The  optimal 
assignment  is 

If  x  ^30  assign  applicant  to  job  A. 

If  30  >  x^24  assign  applicant  to  job  B. 

If  24  >  x  assign  applicant  to  job  C. 

Insert  Table  2  Near  Here 
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i it  consider  N  applicants  and  without  loss  of  generality 
assume  their  tost  soon-  sat  lsf  ios  T(x.)  >T(x  )  >...>T(x .,)  . 

We  may  have  the  following  situations: 

(1)  quota  free  for  all  jobs,  i .  c* .  we  can  accept  as  many  applicants 
as  we  want  for  each  joh. 

(2)  We  want  to  accept  exactly  NA-NH.NC  • • •  NK  applicants  for  Jobs 

A,B,C,...R  respectively.  So  in  this  case  we  assume 

N.+N  +N •  .  .  +N_  *  N. 

ABC  R 

(3)  For  every  job  A,B,C,...R  we  want  to  accept  at  most  N^,  N^,  ... 

respectively.  So  in  this  case  we  assume  N^+Ng+N£+  • 

Before  we  find  optimal  decision  procedure  for  the  above  situations 

we  should  notice  thac  (3)  is  the  most  general  one.  In  situation  (3)  if 

we  have  the  additional  condition  that  N  “N  ”N  ..."  N  =  N  then  (3) 

ABC  R 

specializes  to  (1).  If  N  +N  +...  +N  =  N  then  we  can  also  see  (3) 

A  B  R 

specializes  to  (2).  In  the  following  we  will  consider  situation  (3) 
first  and  discuss  situations  (1)  and  (2)  in  the  corollary  at  the  end 
of  this  section. 

Before  we  develop  an  optimal  decision  procedure  for  situation  (3) 

we  need  some  definitions  and  preliminary  results. 

We  can  illustrate  situation  (3)  as  follows 

C  B  A 

. I...  OOI...  OOI  ...pool 

'  ' - - -  '  v - J  w“  V - - — ' 

N„  circles  N„  circles  N.  circles 
C  B  A 

where  A,B,C...  are  job  categories,  and  the  number  of  circles 


under  each  job  is  the  number  of  openings  for  that  job  category.  When 
we  assign  an  applicant  to  a  job  we  put  his  number  In  the  corresponding 
circle.  For  example,  the  following  picture  represents  our  assigning 


applicant  1  to  job  A,  applicants  2  and  3  to  job  B. 


..QG)©|...6oG)l 

By  theorem  f3)  of  the  previous  section  and  the  assumption  that 

T(x.)>  T(x0)  ...  >T(x„),  we  can  see  that  an  applicant  with  a  higher  score 
1  —  c  —  N 

should  occupy  a  position  to  the  right  of  a  lower  score  applicant.  Thus, 
we  will  only  consider  assignments  that  satisfy  this  condition.  The 
following  assignment  satisfies  this  condition. 

I©@  [00©IQC©IQG)©Q1o6© 

Here  we  have  9  applicants  and  =  3,  Ng  =  4,  N^.  =  2,  =  3  and 

nr*  2- 

We  can  also  put  index  on  the  openings  in  the  way  that  openings  in 
job  A  have  index  1,2, ...N^  and  openings  for  job  B  have  index  N^+l , . . .N^+Ng 
respect ively . 

After  assignment  of  all  applicants  we  need  the  following  definition. 
Definition  Chain  is  a  set  of  consecutive  applicants  such  that  the 
highest  scoring  applicant  in  the  set  occupies  the  first  opening  of  the 
job  he  got.  All  the  job  openings  between  the  one  occupied  by  the 
highest  score  and  the  one  occupied  by  the  lowest  score  (in  the  chain)  are 
all  occupied  by  the  members  of  the  chain.  And  there  are  still  opening(s) 
for  the  Job  that  the  lowest  scored  nppl  leant  In  t  lie  chain  got.  Also, 
there  are  still  opening(s)  for  the  job  just  to  the  right  of  the 
opening(s)  the  highest  scoring  applicant  got,  if  there  are  any. 
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For  example,  in  the  above  picture,  applicant  1  is  a  chain  and  the 
set  of  applicants  2 , 3 , 4 , 5 , 6, 7 , 8  is  also  a  chain. 

From  the  above  definition,  we  can  have  the  following  lemma. 

Lemma  2:  Suppose  wo  have  the  optimal  assignment  of  applicants  1,2,. ..N. 
Let  the  highest  scoring  applicant  in  a  chain  be  i+l**'  applicant.  We 
assume  there  are  K  applicants  in  the  chain  occupying  v+1,  v+2,...v+k 
indexed  position.  If  we  move  j+1,  j+2,...j+!  (where  i+1  <_j+l  <_  j+1  £  i+k, 
i.e.  all  are  in  the  chain)  from  positions  v+1,  v+2,  ...  v+1  to  positions 
v-n+1,  v-n+2  ,  ...  v-n+1  (where  n  is  any  positive  integer),  then  the 
the  total  expected  utility  will  decrease. 

Proof:  We  first  show  that  we  shift  i+1,  i+2 ,  ...  i+1  applicants  from 
v+1,  v+2,  ...v+1  to  v+n+1,  v+n+2,  ...  v+n+1  then  the  total  expected 
utility  will  decrease  or  not  change. 

When  i+1,  i+2,... i+1  applicants  shift  from  v+1,  v+2,... v+1  to  v, 
v+l,...v+!-l  we  can  see  the  total  expected  utility  will  reduce  or  not 
change  because  before  the  shift  we  already  have  optimal  assignment. 

Now  i+2,  i+3,...i+l  occupy  positions  v+1,  v+2 , . . . v+1-1 ,  respectively. 

By  theorem  2  of  the  previous  section  if  we  shift  applicant  i+1, 
i+2,.. .1+1-1  from  positions  v+1,  v+2, ...v+1-1  to  v ,v+l , . . .v+l-2  the  total 
expected  value  will  decrease  or  not  change.  We  can  see  that  shifting 
applicants  i+2,  i+3 , . . . i+1  from  v+1,  v+2, ...v+1-1  to  v,  v+1,. ..v+l-2  will 
reduce  or  not  change  total  expected  utility. 

For  those  who  occupy  positions  (note  we  only  consider  applicants 
i+1,  i+2,...i+k)  less  than  v+1  we  don't  consider  them  because  the  more 
to  the  right  the  less  expected  utility  we  will  get  from  them. 
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Similarly,  if  we  move  i+3,  i+4,...i+£  from  v+1,  v+2,  ...v+£-2 
openings  one  position  to  the  right  the  total  expected  utility  will  not 
increase.  Thus,  we  can  see  that  moving  i+1 ,  1+2,...  i  +  J  from  positions 
v+l,  v+2,  ...  v+t  to  v+1 -n,  v+2-n, . . . v+l-u  will  not  increase  our  total 
expected  utility. 

Next  we  consider  moving  j+1,  J+2,...J+£  applicants  from  v+1,  v+2,... 
v+t  to  positions  v-n+1,  v-n+2,...  v-n+£.  By  the  above  result  and 
Theorem  2  of  the  previous  section  we  can  see  this  action  will  not 
increase  the  total  expected  utility. 

Now  we  are  ready  to  show  optimal  decision  procedure  for  situation 
(3).  The  procedure  is  as  follows: 

We  arrange  the  applicants  according  to  their  test  scores  and  start 
from  the  first  person  (who  has  the  highest  score)  then  the  second, 
third,  etc...  until  the  one  who  has  the  lowest  test  score  using  the 
following  assigning  algoritlim.  We  consider  ith  applicant  with  test  score 
T(x^)  (i=l,2,...N). 

(i)  assign  the  applicant  to  the  job  that  gives  highest  expected 
utility. 

(ii)  if  the  job  in  (i)  is  filled  (i.e.,  filled  by  some  people  who 
have  higher  test  scores  than  T(x^)),  then  assign  i  to  the  same  job  as 
(i-l)th  applicant. 

(iii)  if  the  job  in  (ii)  is  also  filled  we  have  to  consider  two 
possible  ways:  (1)  assign  i  to  the  next  job  opening  that  has  lower 
marginal  utility  than  the  job  (i-1)  applicant  was  assigned  or  (2)  assign 
i  to  the  same  job  as  (i-1)  applicant,  but  now  this  job  accepts  more 
applicants  than  its  limit.  Thus,  we  shift  all  applicants  in  the  chain 
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that  contains  (i-l)th  applicant  one  position  to  the  right.  From  these 
two  ways  we  select  the  one  that  has  higher  expected  utility. 

The  above  decision  procedure  is  optimal  since: 

When  (i)  is  true  we  can  see  immediately  we  have  assigned  applicant 
1,2,...  i  —  1 ,  I  optimally  because  wo  assigned  applicant  1,2,.. .1-1  and 
i  optimally. 

To  consider  (ii)  we  have  to  compare  the  assignment  that  satisfies 
the  condition  that  an  applicant  with  a  higher  score  should  occupy  a 
position  on  the  right.  (This  is  another  way  of  expressing  Theorem  2  in 
the  previous  section). 

We  know  applicant  i  has  to  occupy  an  opening  at  least  to  the 
right  of  the  opening  we  originally  assigned,  or  the  new  assignment 
will  not  increase  our  total  expected  utility. 

Let  the  new  arrangement  satisfy  the  above  conditions.  We  will 
shift  our  original  arrangement  in  steps  to  the  new  arrangement  such 
that  each  step  our  total  expected  utility  will  not  increase.  And 
thus  our  originally  assignment  has  highest  expected  utility. 

We  first  shift  the  last  chain  in  our  original  arrangement  one 
position  to  the  right.  By  (2)  in  Lemma  2,  our  total  expected  utility 
will  not  increase  (note  ith  applicant  does  not  change  job,  although 
he  changes  openings).  Now  we  shift  this  chain  to  the  right  until  ith 
applicant  occupies  the  same  position  as  a  new  arrangement. 

Every  shift,  by  (T>  in  Lemma  2,  will  not  increase  total  expected 
utility.  We  now  shift  all  except  1  to  the  right  until  i-lth  applicant 
occupies  the  position  in  the  new  arrangement.  We  now  move 

applicants  in  the  last  chain,  except  i-1  and  i  to  the  right  until 


28  | 


i-2th  applicant  occupies  the  positions  in  the  new  arrangement  and 
each  step  will  not  increase  our  total  expected  utility,  thus  (ii) 
gets  the  highest  expected  utility. 

It  can  be  shown  that  (lii)  is  optimal  by  exactly  the  same  way 
as  in  (ii) . 

Corollary: 

(1)  For  a  quota  free  situation,  the  optimal  decision  procedure  is 
to  assign  each  applicant  to  the  job  that  gives  highest  expected 
utility. 

(2)  For  situation  (2),  the  optimal  decision  procedure  is  to  assign 
applicant  1,2,...NA  to  job  A,  NA+1,  NA+2,  ...  NA+Nft  to  Job 
R,...,  and  the  last  NR  applicants  to  job  R. 

The  above  corollary  is  a  direct  result  of  our  previous  decision 


procedure. 
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Selections  Considering  Applicant’s  Preference 

In  this  part,  we  take  applicant's  preference  into  consideration. 

Suppose  we  have  two  jobs  and  N  applicants.  Every  applicant  states 

which  job  he  prefers.  I.et  these  two  Jobs  be  A  and  B  and  each  has 

n  and  n  openings.  We  further  assume  n.  +  n  >  N  and  the  decision 
n  b  A  B  — 

maker  know  each  applicant's  preference  between  jobs  A  and  B.  A 
decision  maker  assigns  each  applicant  to  either  job  A  or  B  but  not 
both 

Let  Uj(9)  and  11^(9)  satisfy: 

1^(0)  >  U2(0)  (2) 

V<e)  i  u2'(e)  (3) 

Now  we  define  utility  functions.  Suppose  ith  applicant  prefers 
job  A  to  B.  We  assume 

UiA(0)  “  Ul(6) 

UiB<e)  -  U2(0) 

where  11^(0)  and  11^(0)  are  the  utility  function  if  we  assign  the 
ith  applicant  to  jobs  A  and  B  respectively. 

Similarly,  if  jth  applicant  prefers  job  B  to  A,  then 

Ve>  ■ 

uJB<e)  -  Ul<0) 

Every  applicant  will  have  higher  utility  if  we  assign  him  to  the  job 
be  perferred.  Thus  assumption  (2)  is  very  reasonable.  We  will  see 
the  case  when  (3)  is  not  true  later  in  this  part. 
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A  trivial  special  case  is  when  n  «•  n  ■  N,  then  we  assign 

A  i> 

every  applicant  to  the  job  he  prefers. 

Under  the  assumptions,  the  decision  procedure  that  will  maximize 
the  total  sum  of  expected  utility  is  as  follows: 

Arrange  the  applicants  according  to  T(x  )  >  T(x„)  ...  >  T(x  ). 

i  —  2  —  n 

Starting  from  the  first  to  the  last  applicant,  assign  each  (say 
ith)  applicant  to  the  Job  he  preferred,  if  there  is  no  opening  for 
that  job  (say  already  filled  by  1 ,  2 ,  3  . . .  i  -  1  applicant)  then 
assign  him  to  the  other  job. 

Next  we  prove  the  above  algorithm  maximizes  the  total  expected  utility. 
Suppose  at  the  end  job  A  still  has  openings,  then  we  cannot  move  any 
applicants  from  job  B  to  A.  Those  in  Job  B  prefer  job  R  to  A,  (or  they 
will  be  in  job  A),  thus  moving  them  to  A  will  not  increase  our  total 
expected  utility.  When  job  B  still  has  openings  we  can  prove  optimality 
in  the  same  way. 

Next  suppose  we  assign  j  to  job  A  and  k  to  job  B,  we  show 
exchange  of  Jobs  between  J  and  k  will  not  increase  our  total  expected 
utility.  Without  loss  of  generality  we  assume  T(Xj)  >  T(x^),  and 
thus  j  applicant  prefers  job  A  to  job  B.  If  k  applicant  prefers 
job  B  to  job  A,  then  j  and  k  are  happy  and  we  should  not  exchange 
their  jobs. 

If  k  applicant  also  prefers  Job  A  to  B  then  UjA(o)  “  1Va^*0  “ 

U, (6)  and  U„  (0)  «  U,„(fl)  »  U„(P).  Bv  Theorem  2,  we  have 

1  n  KB  / 

Va(T(Xj)>  +  VB(T(xk))  >  Va(T(xr))  +  VR(T(x j)) 
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Thus  exchange  J  and  k's  job  will  not  increase  total  expected 
utility.  Thus  we  have  optimal  decision  procedure. 

If  wo  assume  llj(C)  and  11,(0)  satisfy: 

Ut(0)  >  U2(0)  (2) 

V(e)  -  V(0)  (3) ' 

and  when  l  prefers  job  A  to  B  then  U  (0)  -  U,  (0)  and  U._(0)  -  U_(B). 

Then  the  optimal  decision  procedure  is  as  follows:  Arrange 
T(xp  >  T(x,)  ...  >_  T(x^),  but  now  starting  from  Nth  applicant  (the 
one  who  has  lowest  test  score)  till  the  first  applicant,  assign  any 
applicant  to  the  Job  he  preferred,  if  no  opening  there  then  assign 
him  to  the  other  Job.  The  proof  is  the  same  as  previous  case. 

In  this  case  we  honored  lower  score  applicants  first.  This 
may  not  be  acceptable  in  moat  of  the  real  world  cases. 

Another  situation  is  that  -  U2'(0),i.e.  1^(0)  and  l'.,(0)  differ 

by  a  constant.  Optimal  decision  procedure  is  started  from  any  one 
applicant  (each  time  arbitrary  choose  next  applicant),  if  the  Job 
ho  preferred  still  has  opening  assign  him  to  there  if  no  opening, 
assign  him  to  the  other  Job.  We  can  sec  in  this  case  a  lot  of  arrange¬ 
ments  have  maximum  expected  utility. 


Table  1.  The  expected  utilities  for  selection  for  two  sample  sizes  at 
the  various  points  of  the  test  scores. 


Expected 


Score  n  =  105 


0 

0.057 

2 

0.081 

4 

0.112 

6 

0.154 

8 

0.206 

10 

0.269 

12 

0.342 

14 

0.422 

16 

0.507 

18 

0.591 

20 

0.672 

22 

0.744 

24 

0.807 

26 

0.859 

20 

0.899 

30 

0.930 

32 

0.952 

34 

0.968 

36 

0.979 

Ut ilit ies 


n  -  10 

0.457 

0.471 

0.488 

0.507 

0.529 

0.555 

0.585 

0.618 

0.653 

0.689 

0.723 

0.752 

0.773 

0.788 

0.796 

0.800 

0.800 

0.799 

0.796 
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1  DR.  RALPH  DUSEK 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22 j 33 

1  Dr.  Kyron  Fischl 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 333 

1  Dr.  Beatrice  J.  Farr 

Army  Research  Institute  (PERI-OK) 

5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Milt  Maicr 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

1  Dr.  Harold  F.  O'Neil,  Jr. 

ATTN:  PERI-OX 

5001  EISENHOWER  AVENUE 

ALEXANDRIA,  VA  22333 

1  Dr.  Robert  Ross 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Sasmor 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
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Ari:iy 


1  Director,  Training  Development 
U,:;,  Army  Administration  Center 
ATTN:  Dr.  Sherrill 
Ft.  Benjamin  Harrison,  IN  46? 13 

1  Dr.  Frederick  Stelnheiser 
1).  S.  Army  R-'surch  Institute 
50^1  Eisenhower  Avenue 
Alextndrio,  VA  PP33 3 

1  Dr  .  Joseph  Uard 

U.S.  Arny  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 J 33 


Air  Force 


1  Air  Force  Human  Resources  Lab 
AFMRL/PED 

Hrooks  AFP,  IX  7  ”230 

1  Air  University  Library 
AUL/LSF.  7L/113 
fax we 11  AFP,  AL  <C11? 

1  Dr  .  Pti  i  1  i  p  De  Loo 
AFHHL/TT 

Lowry  AFi),  CO  802j0 

1  DR.  G.  A.  ECK3TKA1ID 

AFHRL/AS 

WRIGHT-PATTERSON  AFP,  OH  15133 

1  Dr.  Genevieve  Haddad 
Program  l.muger 
Life  Sciences  Directorate 
AF03R 

Polling  AFB,  DC  20332 

1  CDIi.  MERCER 

CfIF.T  LIAISON  OFFICER 
AFHRL/FLYING  TRAINING  D1V. 
WILLIAMS  AFB,  AZ  85 221 

1  Dr.  Ross  L.  Morgan  (AFHRL/ASR) 
Wright  -Patterson  AFI' 

Ohio  45Mj3 

1  Dr.  Roger  Pennell 
AFI1RL/TT 

Lowry  AFB,  CO  80230 

1  Personnel  Analysis  Division 
HQ  USAF/DPXXA 
Washington,  DC  20J30 

1  Research  Branch 
AFMPC/DPHYP 

Randolph  AFB,  TX  78118 

1  Dr.  Malcolm  Ree 
AFHRL/PED 

Brooks  AFB,  TX  782 35 
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Air  Force 


Marines 


1  Dr.  Marty  Rockway  ( AFHRL/TT) 
Lowry  AFB 
Colorado  30230 

1  Jack  A.  Thorpe,  Capt,  USAF 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 


1  H.  William  Greenup 

Education  Advisor  (E03O 
Education  Center,  MCDEC 
Quantico,  VA  2213^ 

1  Director,  Office  of  Manpower  Utilization 
HQ,  Marine  Corps  (MPU) 

BCB,  Bldg.  2009 
Quantico,  VA  221 3*4 


1  Brian  K.  Waters,  LC0L,  USAF 
Air  University 
Maxwell  AFB 
Montgomery,  AL  -6112 


1  DR.  A.L.  SLAFK0SKY 

SCIENTIFIC  ADVISOR  (CODE  RD-1 ) 
HQ,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20380 
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CoastGuard 


Other  DoD 


1  Mr.  Richard  Lanterman 

PSYCHOLOGICAL  RESEARCH  (G-P-1/62) 
U.S.  COAST  GUARD  HQ 
WASHINGTON,  DC  20590 

1  Dr .  Thomas  Warm 

U.  S.  Coast  Guard  Institute 
P.  0.  Substation  It? 

Oklahoma  City,  OK  73159 


12  Defense  Docunentation  Center 
Cameron  Station,  Bldg.  5 
Alexandria,  VA  22319 
Attn:  TC 

1  Dr.  Dexter  Fletcher 

ADVANCED  RESEARCH  PROJECTS  AGENCY 
1900  WILSON  BLVD. 

ARLINGTON,  VA  22209 

1  Dr.  William  Graham 
Testing  Directorate 
IIEPCCM 

Ft.  Sheridan,  IL  600 j7 

1  Military  Assistant  for  Training  and 
Personnel  Technology 

Office  of  the  Under  Secretary  of  Defense 
for  Research  &  Engineering 
Room  3D  129.  The  Pentagon 
Washington,  DC  2C-301 

1  MAJOR  Wayne  Sellman,  USAF 

Office  of  the  Assistant  Secretary 
of  Defense  ( MftA&L) 

3B930  The  Pentagon 
Washington,  DC  20301 
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Civil  Govt 


Civil  Govt 


1  Dr.  Susan  Chipman 
Basic  Skills  Program 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  William  Gorham,  Director 
Personnel  R&D  Center 
Office  of  Personnel  Managment 
1900  E  Street  NW 
Washington,  DC  20415 


1  Dr.  Thomas  G.  Sticht 
Basic  Skills  Program 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Vern  W.  Urry 

Personnel  R4D  Center 
Office  of  Personnel  Managmcnt 
1900  E  Street  NW 
Washington,  DC  20415 


1  Dr.  Frank  Withrow 

U.  S.  Office  of  Education 
400  6th  Street  SW 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Memory  4  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 


1  Dr  .  Arthur  Melmed 

National  Intitute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Andrew  R.  Kolnar 
Science  Education  Dev. 
and  Research 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Lalitha  P.  Sanathanan 

Environmental  Impact  Studies  Division 
Argonne  National  Laboratory 
9700  S.  Cass  Avenue 
Argonne,  IL  60439 

1  Dr.  Jeffrey  Schiller 

National  Institute  of  Education 
1200  19th  St.  NW 
Washington,  DC  20203 


1  Dr.  Joseph  I.  Lipson 

Division  of  Science  Education 
Room  W-638 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  John  Mays 

National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 
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’Jon  Govt 


1  Dr.  Earl  A.  Alluisi 
KG,  AFHRL  (AFSC) 

Prooks  AFB,  TX  73239 

1  Dr.  Erling  0.  Anderson 
University  of  Copenhagen 
Studiestraedt 
Copenhagen 
DENMARK 

1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
C  inberra  ACT  2600,  Australia 

1  Dr.  Alan  Bad del ey 

Medical  Research  Council 

Applied  Psychology  Unit 
19  Chaucer  Road 
Cambridge  CB2  2EF 
ENGLAND 

1  Dr  .  Isaac  Pejar 

Educational  Testing  Service 
Princeton,  NJ  03450 

1  Dr.  Warner  Birice 
Streitkraefteamt 
Rosenberg  5300 
Bonn,  West  Germany  D-5300 

1  Dr.  R.  Darrel  Bock 

Department  of  Education 
University  of  Chicago 
Chicago,  IL  CO637 

1  Dr.  Nicliolas  A.  Bond 
Dept,  of  Psychology 
Sacramento  State  College 
600  Jay  Street 
Sacramento,  CA  95319 

1  Dr.  David  G.  Dowers 

Institute  for  Social  Research 
University  of  Michigan 
Ann  Arbor,  HI  48106 


Non  Govt 


1  Dr  .  Robert  Brennan 

American  College  Testing  Programs 

P.  0.  fox  168 

Iowa  City.  IA  52240 

1  DR.  C.  VICTOR  BUNDERSOH 
WICAT  INC. 

UNIVERSITY  PLAZA ,  SUITE  10 
1160  SO.  STATE  ST. 

OREM,  UT  34057 

1  Dr.  John  P.  Carroll 
Psyclwnetr ic  Lab 
Utiiv.  of  No.  Carolina 
Davie  Hall  013A 
Chapel  Hill,  NC  27514 

1  Charles  Myers  Library 
Livingstone  House 
Livingstone  Road 
Stratford 
London  F 15  2LJ 
ENGLAND 

1  Dr.  John  Chiorini 
Li tton-Mel Ionics 
box  1286 

Springfield,  VA  22 151 

1  Dr.  Kenneth  E.  Clark 

College  of  Arts  i.  Sciences 
University  of  Rochester 
River  Campus  Station 
Rochester,  NY  14627 

1  Dr.  Norman  Cliff 
Dept,  of  Psycliology 
Univ.  of  So.  California 
University  Park 
Los  Angeles,  CA  90007 

1  Dr.  William  Coffman 
Iowa  Testing  Programs 
University  of  Iow.i 
Iowa  City,  IA  52242 
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Non  Govt 


Non  Govt 


1  Dr.  Meredith  Crawford 

Department  of  Engineering  Administration 
George  Washington  University 
Suite  805 

2101  L  Street  N.  W. 

Washington,  DC  20037 

1  Dr.  Hans  Cronbag 

Education  Research  Center 
University  of  Leyden 
Boerhaavelaan  2 
Leyden 

The  NETHERLANDS 

1  Dr .  Bnmanuel  Donchin 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  MAJOR  I.  N.  EVONIC 

CANADIAN  FORCES  PERS.  APPLIED  RESEARCH 
1107  AVENUE  ROAD 
TORONTO,  ONTARIO,  CANADA 

1  Dr .  Leonard  Feldt 

Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City,  TA  52242 

1  Dr.  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.0.  Pox  168 

Iowa  City,  IA  52240 

1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  KD  20850 

1  Dr.  Gerhard t  Fischer 
Liebigasse  5 
Vienna  1010 
Austria 

1  Dr.  Donald  Fitzgerald 

University  of  New  England 
Armidale,  New  South.  Wales  23 51 
AUSTRALIA 


1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 
Suite  900 

4330  East  West  Highway 
Washington,  DC  20014 

1  Dr.  John  R.  Frederiksen 
Bolt  Beranek  A  Newman 
50  Moulton  Street 
Cambridge,  HA  021 38 

1  DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  15213 

1  Dr.  Ross  Greene 
CTB/McGraw  Hill 
Del  Monte  Research  Park 
Monterey,  CA  93940 

1  Dr.  Alan  Gross 

Center  for  Advanced  Study  in  Education 
City  University  of  New  York 
New  York,  NY  10036 

1  Dr.  Ron  llumbleton 
School  of  Education 
University  of  Massechusetts 
Amherst ,  MA  01002 

1  Dr.  Chester  Harris 
School  of  Education 
University  of  California 
Santa  Barbara,  CA  93106 

1  Dr.  Lloyd  Humphreys 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Library 

HumRRO/Western  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 
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Non  Govt. 


1  Dr.  Steven  Hunki 

Department  of  Education 
University  of  Alberta 
Edmonton,  Alberta 
CANADA 

1  Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  931 05 

1  Dr.  Huynh  Huynh 

Department  of  Education 
University  of  South  Carolina 
Columbia ,  SC  29308 

1  Dr.  Carl  J.  Jensema 
Gnllaudet  College 
Kendall  Green 
Washington,  DC  2C00? 

1  Dr.  Arnold  F.  Kanarick 
Honeywell,  Inc. 

2600  Ridgeway  Pkwy 
Minneapolis,  MN  55413 

1  Dr.  John  A.  Keats 

University  of  Newcastle 
Newcastle,  New  South  Wales 
AUSTRALIA 

1  Mr.  Marlin  Kroger 
1117  Via  Goleta 

Palos  Verdes  Estates,  CA  90274 

1  LCOL.  C.R.J.  LAFLEUR 
PERSONNEL  APPLIED  RESEARCH 
NATIONAL  DEFENSE  HQS 
101  COLONEL  BY  DRIVE 
OTTAWA,  CANADA  K1A  0K2 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
University  of  Illinois 
Champaign,  IL  61820 


Non  Govt 


1  Faculteit  Sociale  V/e  ten  sc  happen 
Ri jksuniversiteit  Groningen 
Oude  Boter ingestraat 
Groningen 
NETHERLANDS 

1  Dr.  Robert  Linn 

College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Frederick  M.  Lord 

Educational  Testing  Service 
Princeton,  NJ  035*10 

1  Dr .  Gary  Marco 

Educational  Testing  Service 
Princeton,  NJ  08450 

1  Dr.  Scott  Maxwell 

Department  of  Psychology 
University  of  Houston 
Houston,  TX  77025 

1  Dr  .  Sam  Ma  yo 

Loyola  University  of  Chicago 
Chicago,  IL  60601 

1  Dr.  James  A.  Paulson 

Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

1  MR.  LUIGI  PETRULLO 

2431  N.  EDGEWOOD  STREET 
ARLINGTON,  VA  22207 

1  DR.  STEVEN  M.  PINE 
4950  Douglas  Avenue 
Golden  Valley,  MN  55*116 

1  DR.  DIANE  M.  RAMSEY-KLEE 

R-K  RESEARCH  &  SYSTEM  DESIGN 
39*17  RIDGEMONT  DRIVE 
MALIBU,  CA  90265 
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Non  Govt  Non  Govt 


,  1  MIN.  RET.  M.  RAUCH 
P  II  4 

BUNDESM INISTERIUM  DER  VERTEIDIGUNG 
POSTFACH  161 
53  BONN  1,  GERMANY 

1  Dr.  Peter  B.  Read 

Social  Science  Research  Council 
605  Third  Avenue 
New  York,  NY  10016 

1  Dr.  Mark  D.  Reckase 

Educational  Psychology  Dept. 
University  of  Missour i-Colunbia 
12  Hill  Hall 
Colimbia,  MO  65201 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

1  Dr.  Leonard  L.  Rosenbaum,  Chairman 
Department  of  Psychology 
Montgomery  College 
Rockville,  MD  20350 

1  Dr.  Ernst  Z.  Rothkopf 
Pell  Laboratories 
600  Mountain  Avenue 
Murray  Hill,  NJ  07974 

1  Dr.  Donald  Rubin 

Educational  Testing  Service 
Princeton,  NJ  03450 

1  Dr.  Larry  Rudner 
Gallaudet  College 
Kendall  Green 
Washington,  DC  20002 

1  Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 


1  PROF.  FUMIKO  SAMEJIMA 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  TENNESSEE 
KNOXVILLE,  TN  37916 

1  Dr.  Kazao  Shigemasu 
University  of  Tohoku 
Department  of  Educational  Psychology 
Kawauchi,  Sendai  982 
JAPAN 

1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  1 1 A ,  Yale  Station 
Hew  Haven,  CT  06520 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94305 

1  Dr.  Hnriharan  Swaminathan 

Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

1  Dr.  Brad  Sympson 

Office  of  Data  Analysis  Research 
Educational  Testing  Service 
Princeton,  NJ  03541 

1  Dr.  Kikumi  Tatsuoka 

Computer  Based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbana,  1L  61801 
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Non  Gov  t 


Non  Govt 


1  Dr  .  Maurice  Tatsuoka 

Department  of  Educational  Psychology 
University  of  Illinois 
Champaign,  IL  6 1  i*0 1 

1  Dr.  David  Thissen 

Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

1  Dr .  Robert  Tsutakawa 
Dept,  of  Statistics 
University  of  Missouri 
Colunbia,  MO  65201 

1  Dr.  J.  Uhl nner 

Perceptronics,  Inc. 

6271  Variel  Avenue 
Woodland  Hills,  CA  91364 

1  Dr.  Howard  Wainer 

Bureau  of  Social  Science  Research 
1990  M  Street,  N.  Vi. 

Washington,  DC  20036 

1  DR.  THOMAS  WALLSTEN 

PSYCHOMETRIC  LABORATORY 
DAVIE  HALL  013A 
UNIVERSITY  OF  NORTH  CAROL 
CHAPEL  HILL,  NC  27514 

1  Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455 

1  DR.  SUSAN  E.  WHITELY 
PSYCHOLOGY  DEPARTMENT 
UNIVERSITY  OF  KANSAS 
LAWRENCE,  KANSAS  66044 

1  Dr.  Wolfgang  Wildgrube 
Streitkraefteamt 
Rosenberg  5300 
Bonn,  West  Germany  D-5300 


1  Dr.  J.  Arthur  Woodward 
Department  of  Psychology 
University  of  California 
Los  Angeles,  CA  90024 

1  Dr  .  Robert  Uoud 

School  Examination  Department 

University  of  London 

66-72  Gower  Street 

London  WCIE  CEE 

ENGLAND 

1  Dr.  Karl  Zinn 

Center  for  research  on  Learning 
and  Teaching 
University  of  Michigan 
Ann  Arbor,  MI  49104 


